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ABSTRACT 



The paper describes the system of rater comments used in an 
American College Testing (ACT) Program English language test designed for use 
with Japanese secondary school students learning English as a foreign 
language. The writing portion of the test consists of two short essays, each 
scored by two native speakers of North American English, using a six-point 
rubric. In addition to giving the essay points, the first reader of each 
essay may select four comments from a standardized list concerning the 
essay's fluency and development, organization, vocabulary, and language 
elements. Mechanics and spelling are not addressed. Raters are trained to 
look for meaning in the work and to see past mechanical problems in the 
writing. Rater comments are to provide (1) praise and (2) suggestions for 
improvement. Examples of comments in each area are included. Advantages of 
this approach include the time saved by teachers in providing a comment by 
number, and the ease with which standardized comments can be understood by 
students. Contains one reference. (MSE) 
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What are Rater Comments? 



Rater Comments are comments about writing that raters can choose 
from a list of available comments. The purpose of these Rater Comments is 
^ to give students more information on the results of their writing assessment 

than simply numerical scores. The Comments are not given to justify the 
^ student scores but rather are designed to give additional narrative feedback 

g to students that they and their teachers can use to build student confidence 

and to improve writing skills in the classroom. 

For the past two years, ACT has been working on a contract with the 
Benesse Corporation of Japan to produce an English language test to be used 
for Japanese secondary students who are learning English as a Foreign 
Language. The purpose of the Benesse English Test is to test English — 
basically American English — in an authentic and communicative way. The 
Benesse test contains sections on listening, reading, and writing. The 
listening portion is based on authentic American speech patterns, and the 
reading portion is based on authentic reading materials. The direct writing 
assessment asks the students to write two short essays. Students have 10 to 12 
minutes to write each essay. Since all three sections of the test are designed to 
fit into two class periods in Japanese junior and senior high schools, the 
writing section must be completed in a total of 20 or 25 minutes, depending 
on which level of test a student is taking. 

Each essay is scored by two native speakers of American English using a 
6-point rubric. Scoring is holistic but based on the traits listed within the 
rubric, namely fluency or development, organization, vocabulary, and 
language elements. In addition to giving a score of 1 to 6 points, the first rater 
of each essay gives it four comments selected from the list of comments. 

What are the underlying assumptions? 

Rater Comments have been used previously for several of ACT's other 
writing assessments such as the 8th and 10th grade assessments and the 
portfolio assessment, though the list was modified and added to for this EFL 
writing assessment. Some of the main assumptions that underlie the Rater 
Comments are: 



o 




1. Raters try to convey perceptions of achievement and areas needing 
improvement specific to each paper. When giving comments to a 
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paper. Raters are asked to think of what they would like to say to a 
student if they were able to have a one-to-one conference with that 
student — ^what they might say about the strengths of the writing and 
what they might say that would help the student revise this paper or 
improve the next one. 

2. The tone of the response is supportive and facilitative. We want to 
build the student’s confidence, not undermine it. 

3. Responses are as explicit as possible and are related to mformation 
shared in the classroom as much as possible. 

4. Raters are of necessity responding to "final" drafts, but, with their 
choice of comments, they can encourage revision and different 
approaches and behaviors in future work. On the Benesse test, for 
example, students will get their papers back with these comments so 
they can be used for revision purposes in the classroom 

5. Raters use four comments for each paper. They are asked to use at least 
one positive and one improvement comment. The other two 
comments are left to raters’ discretion. We’ve learned through 
feedback on other projects that all students, no matter how 
experienced, need positive encouragement on their writing. Then, too, 
even the best student writers want to be shown how they can improve, 
and, of course, the less experienced writers need suggestions for 
improvement. 

How are Rater Comments used? 

The Rater Comments come from the same four areas that are 
measured by the rubric: fluency and development, organization, vocabulary, 

and language elements. Notice that mechanics and spelling are not included 
among these comments The rubric considers these things only to the extent 
that they interfere with meaning — to the extent that the errors are global 
rather than local errors, to use terms used in TheGooficon by Burt and 
Kiparsky. 

After ACT receives the student papers from Japan, the next task is to 
prepare raters, most of whom are accustomed to working with native speaker 
writing. Raters learn what to look for (or NOT to look for) when rating 
papers written in a second language We give the raters a crash course in 
ESL/EFL with lots of examples from student writing, prior to actually training 
them on sets of papers representing the various score points. 

The main point of our training is to teach raters how to look for 
meaning, to try to understand what has been written, even when it is 
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confusing to them in some way. For example, it is an easy thing to train 
raters to read words that are spelled with "r"s and "l"s that are reversed, and 
they easily learn to read "green grass" even though the paper says "green 
glass," or to see that sometimes Japanese students will write the extra, 
epenthesized vowel that they often say when pronoimcing an English word 
with a consonant cluster. One of our favorite words is "stores." It looks like 
the English word "stores," but when you see the context you can tell that it 
means "stress." 

Raters also need to learn that students may not be accustomed to 
English punctuation and that they may write sentences in a "listing" format 
rather than in a paragraph format. We mention the difficulties that nearly all 
speakers and writers of English as a second or foreign language have with 
verb forms, pronoims, prepositions, and articles. We point out a few specific 
words that Japanese students may use in their essays, such as the word 
"shinkansen" which means "bullet train." We discuss how hard it is to use 
the appropriate tone and register in a language that is not your LI, and we 
show how "organization" is often evidenced by the use of simple 
coordinating or subordinating conjunctions or perhaps by simple conjunctive 
adverbs such as "first, " "second," and "third". 

With this backgroimd, we then introduce raters to actual student 
papers, practice scoring using the rubric, and then practice giving comments 
to the papers. 

The comments fall under the four categories mentioned before. 

Within each category are two kinds of comments; praise and suggestions for 
improvement. Since regardless of the quality of the paper, the requirement is 
that at least one of the four comments must be praise and at least one must be 
a suggestion for improvement, the list of comments includes a variety of 
comments. This means, for example, that the list must include some praise 
comments that could be used for a low-level paper with many problems as 
well as suggestions for improvement that could be given to a high-level 
paper with few problems. 

What do Rater Comments look like? 

Below are just a few examples of the kinds of comments that can be 
used, listed in the four categories mentioned above; 



Fluency and Development 



Praise 

You use clear reasons to support yotir opinions. 
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Su ggestions for Improvement 

Your writing would be more interesting if you added related details 
about what you see, hear, taste, or smell. 



Praise 



Vocabulary 



The words you use are usually accurate. 



Su ggestions for Improvement 

Greater variety in your word choice would make your writing more 
interesting. 



Praise 



Organization 



The transitions in your writing show clear connections among ideas. 



Su ggestions for Improvement 

Following your ideas was sometimes difficult. 



Praise 



Language Elements 



Your use of a variety of sentence structures helps make yovir writing 
interesting. 



Su ggestions for Improvement 

You might consider combining several ideas into one sentence. 



Additional Uses for Rater Comments 



We use the Rater Comments in large-scale assessment, but we believe 
that the practice of giving comments from a list is also adaptable to various 
kinds of writing response. For example, comments could be useful for 
writing that is done for university, community college, or lEP placement 
testing so that a student, as well as the teacher, would receive some feedback 
from the writing based on what the teachers in the program feel is important. 
In a classroom setting, a teacher could devise his or her own list of comments 
based on what that teacher would like to focus on. Or a teacher could develop 
a comprehensive list of comments but choose a particular group of comments 
to focus on for a particular type of writing or a particular assignment. 

One advantage for the teacher is the time-saving factor of writing down a 
number rather than a prose comment. After working with a list of comments 
for a while, one begins to know, without looking, which number goes with 
which comment. An advantage for students is Qiat teachers can discuss this 
focused list of comments with students so that students understand what 
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each comment means. One size never fits all, but research indicates that ESL 
students sometimes don't vmderstand the comments that their teachers write 
in the margins of their papers, or that sometimes they can't read the teacher's 
handwriting. A list of comments could also be used for peer feedback among 
students. So using a list of comments to give at least part of the response to 
ESL/EFL writing in various settings is something to consider. 

Reference 
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